A Diagrammatic Reasoning Architecture: Design, Implementation and Experiments

This paper explores the idea that the cognitive state during problem solving diagrams is bi-modal, one of whose components is the traditional predicate-symbolic representation composed of relations between entities in the domain of interest, while a second component is an internal diagrammatic representation. In parallel with the operators in the symbolic representation that are based on symbol matching and inferencing, there is a set of operators in the diagrammatic component that apply perceptions to the elements of the diagram to generate information. In addition there is a set of diagram construction operations that may modify the diagram by adding, deleting and modifying the diagrammatic elements, in the service of problem solving goals. We describe the design of the diagrammatic component of the architecture, and show how the symbolic and diagrammatic modes collaborate in the solution of a problem. We end the paper with a view of the cognitive state as multi-modal, in consonance with our own phenomenal sense of experiencing the world in multiple modalities and using these senses in solving problems. Bimodality of the Problem State in Diagrammatic Reasoning The standard account of cognitive state in both AI and cognitive science involves state representations that are sentences composed of predicates that describe relations that hold between various entities in the domain of interest. Cognitive state changes as a result of applying operators that change parts of the descriptions into other descriptions. For example, an agent contemplating a Blocks World situation would have state representations of 1 This symbolic representation framework applies to both logicist traditions in AI as well as those schools, such as frame and script theories, that situate themselves as alternatives to logicism. The main idea is that all of them view the agent as having knowledge that is expressed in terms of symbols that stand for individuals and relations between them. The differences between the logicists and logic skeptics are about content theories and inference, not about the predicate structure of the representation. the form ON(A, B) ^ ON(C, D), and the state change corresponding to removing block A to the table would be accomplished by Add & Delete operators that change the description to ON(A, Table) ^ ON(C, D). If for his problem solving goals he needs to know the Left-of or Right-of relation between the blocks at various times, as each operation is carried out a number of Left-of and Right-of relations would need to be added and deleted. For even a modest number of blocks, the number of such relations to be updated can become significant in number. Now suppose the agent is now solving the same Blocks world problem while looking at actual blocks. As he moves Block A to the table, there is really no need for an internal set of Add and Delete rules relating to relations between the blocks. As the movement is done, he can simply look at the blocks and note the relations that are relevant for that stage in problem solving. The problem state can now be conceived as bimodal, a part that corresponds to traditional predicate-symbolic representations, and another part that is spatial and available for perception. Extraction of information such as Left-of(B,C) can be thought of as applying a perception operator to the spatial component of the problem state as a parallel to the application of symbolic rule-based operators to the symbolic component of the problem state. Use of diagrams in problem solving is similar. When we have a diagram of a situation, all spatial consequences of an operation, such that of adding or deleting a diagrammatic element or changing the spatial properties of the elements, do not need to be inferred by application of rules to the symbolic component, but obtained as needed from the diagram by applying perception or direct measurement operations on the diagram. The diagram automatically encodes certain consequences of changes, such as emergent objects and emergent relations. When two roads are diagrammed by two curves in a diagram and they intersect, the intersection point and the segments are emergent objects of representational interest. Similarly, when a block A is placed to the left of B which is already to the left of C, an emergent relation is that A is to the left of C. Perception picking up the spatial consequences of changes has been called a free ride (Shimojima, 1996) and is one of most important advantages of a diagrammatic representation. Diagrams do not have to be external. In a good deal of human problem solving people experience mental images similar to a diagram. Of course internal diagrams are subject to short term memory limitations, but phenomenologically, problem solving has a character similar to that when external diagrams are involved. That is, they apply what they experience as internal perception and extract some information that they may combine with linguistically couched information to make further inferences. In these cases, the problem state is completely internal and bimodal. We have been developing an architecture for diagrammatic reasoning that is intended to support problem solving with such bimodal problem states. The linguistic/symbolic mode supports knowledge representations and inferences that are traditional in AI, while the diagrammatic mode supports a representation that captures the spatial aspects of the configuration of objects in a diagram, and an openended set of perceptions that can add information to the symbolic component of the problem state. By the same token, there are operators that can modify the diagram in various ways to capture the effects of intended operations. Problem solving is goal-driven, as in cognitive architectures such as Soar. Problem solving proceeds opportunistically, whichever mode can help in the solution of a subtask makes the contribution, and the problem state changes. This process repeats until the problem is solved (assuming it is solvable given the knowledge). The goal of this paper is to outline our architecture, detail some aspects of the diagrammatic representation, and give an example of its use.